One-on-one comparison between qCSI and NEWS scores for mortality risk assessment in patients with COVID-19

Abstract Objective To compare the predictive value of the quick COVID-19 Severity Index (qCSI) and the National Early Warning Score (NEWS) for 90-day mortality amongst COVID-19 patients. Methods Multicenter retrospective cohort study conducted in adult patients transferred by ambulance to an emergency department (ED) with suspected COVID-19 infection subsequently confirmed by a SARS-CoV-2 test (polymerase chain reaction). We collected epidemiological data, clinical covariates (respiratory rate, oxygen saturation, systolic blood pressure, heart rate, temperature, level of consciousness and use of supplemental oxygen) and hospital variables. The primary outcome was cumulative all-cause mortality during a 90-day follow-up, with mortality assessment monitoring time points at 1, 2, 7, 14, 30 and 90 days from ED attendance. Comparison of performances for 90-day mortality between both scores was carried out by univariate analysis. Results From March to November 2020, we included 2,961 SARS-CoV-2 positive patients (median age 79 years, IQR 66–88), with 49.2% females. The qCSI score provided an AUC ranging from 0.769 (1-day mortality) to 0.749 (90-day mortality), whereas AUCs for NEWS ranging from 0.825 for 1-day mortality to 0.777 for 90-day mortality. At all-time points studied, differences between both scores were statistically significant (p < .001). Conclusion Patients with SARS-CoV-2 can rapidly develop bilateral pneumonias with multiorgan disease; in these cases, in which an evacuation by the EMS is required, reliable scores for an early identification of patients with risk of clinical deterioration are critical. The NEWS score provides not only better prognostic results than those offered by qCSI at all the analyzed time points, but it is also better suited for COVID-19 patients. KEY MESSAGES This work aims to determine whether NEWS is the best score for mortality risk assessment in patients with COVID-19. AUCs for NEWS ranged from 0.825 for 1-day mortality to 0.777 for 90-day mortality and were significantly higher than those for qCSI in these same outcomes. NEWS provides a better prognostic capacity than the qCSI score and allows for long-term (90 days) mortality risk assessment of COVID-19 patients.


Background
In its initial stages back in February-March 2020, the current coronavirus disease 2019 (COVID- 19) pandemic was a global shock to health systems all around the world [1]. In the course of time, the scientific community has developed procedures to adequately screen and manage the huge number of patients generated by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [2,3].
Over the course of the pandemic, several challenges have been overcome in record time, remarkably, early virus detection techniques (antibody and antigen testing) and mass vaccination [4,5]. Nevertheless, the early identification of patients at high-risk of clinical deterioration is still being refined. Initially, patient categorization was complex and early warning scores already implemented in health systems were used, even if they were not specifically designed for COVID-19, as was the case for instance of the National Early Warning Score (NEWS) [6,7].

Importance
Throughout the pandemic, specific scores have been developed to assess disease severity in COVID-19 patients [8][9][10]. The best performing scores have in common the use of laboratory or imaging variables (C-reactive protein, urea, leukocytes, chest computed tomography) not always available in certain healthcare environments [11,12]. Therefore, new simple and easily applicable scores capable of providing relevant risk discrimination capacity have been developed, such as the quick COVID-19 Severity Index (qCSI) [13].
COVID-19 causes an increase in primary care centre consultations, ambulance transfers, emergency department visits, hospital and intensive care unit admissions, and finally, an increase in unexpected mortality. Severe cases present with bilateral pneumonias accompanied by multiorgan disease in which the ventilatory function is especially compromised [14]. Under such circumstances, the early identification of cases at risk of deterioration in the short-term is critical for the strategic management of the pandemic. It is necessary to objectively prioritize those patients with higher probability of survival amongst those at higher risk; at this crossroads, early warning scores can help in the decision-making process [15,16]. Both the qCSI and the NEWS scores include, with different weights, parameters related to the ventilatory function (respiratory rate, oxygen saturation and supplemental oxygen administration), which is critical in the early detection of the risk of deterioration in patients with COVID-19 [7,17,18]. Until now, however, no direct comparison has been done between both scores. This comparison seems mandatory since NEWS is considered the gold standard of early warning scores, and such comparison could determine the added value of the new score.

Goals of this investigation
The present study aims to compare the predictive values of the qCSI and the NEWS scores for 90-day mortality amongst patients with COVID-19 transferred to the emergency department by emergency medical services

Study population
The inclusion criteria were as follows: Adult patients (>18 years) with COVID-19 infection confirmed by SARS-CoV-2 test (polymerase chain reaction) and transferred by ambulance to EDs. The exclusion criteria included: patients under 18 years of age, without analytical confirmation of infection, or cases in which the lack of any variable impeded the estimation of the scores analyzed.

Study design
This was a multicenter retrospective cohort study analyzing health data from March to November 2020, from an overall reference population of 1,166,408 inhabitants. The study was carried out in the provinces of Palencia, Salamanca, Segovia and Valladolid (Spain) with the participation of 61 ambulance services and EDs from 8 hospitals (three tertiary university hospitals and five general district hospitals). Both the Emergency Medical Services (EMS) and the hospitals are managed by the Public Health System of Castilla-Le on (SACYL), the principal health operator.
The study was approved by the local institutional research review board of Rio Hortega Hospital (PI 138/ 20) and conformed to the Declaration of Helsinki principles. The institutional review board granted a waiver of the obligation to collect consent from study participants due to the use of unidentified subjects.

Outcome
The outcome was cumulative all-cause mortality during a 90-day follow-up, with mortality assessment at the following monitoring time points: 1, 2, 7, 14, 30 and 90 days from ED attendance. All monitoring time points were also cumulative, i.e. deaths registered at a particular point also include those registered at the preceding ones, e.g. 14-day mortality includes also patients of the preceding time points 1, 2, 3 and 7 days.

Predictors and data abstraction
Epidemiological data (sex, age, rural or urban area and nursing home origin) and clinical covariates (respiratory rate, oxygen saturation, systolic blood pressure, heart rate, temperature, level of consciousness and use of supplemental oxygen) were collected by an emergency registered nurse at the triage emergency room, so these variables were blinded to the clinical investigators.
The Connex V R Vital Signs Monitor (Welch Allyn, Inc., Skaneateles Falls, NY) was used to measure blood pressure, heart rate, temperature and oxygen saturation. The respiratory frequency was measured by direct observation of the respiratory cycles for 30 s; in case of irregular breathing or extreme ranges, it was measured by direct auscultation for 1 min. The level of consciousness was assessed by means of the Glasgow Coma Scale; a score of less than 15 was considered alteration of the level of consciousness. Finally, the percentage of oxygen was evaluated by means of the fraction of inspired oxygen supplied, from which the litres per minute of oxygen administered were calculated.
Ninety days from the first care, an independent clinical investigator from each hospital reviewed the patient's electronic medical record and collected mortality data, destination (discharge on site, admission for hospitalization or intensive care unit) and 17 categories of comorbidities necessary to calculate the Charlson comorbidity index.
The NEWS and qCSI scores were calculated using vital sign data and clinical observations in accordance with the Royal College of London standards for NEWS (note that NEWS or NEWS2 were used indistinctly, and both refer to NEWS2) [19,20] and Haimovich et al. [13] for qCSI (see Supplementary Data S1 and S2). In particular, values for both scores were calculated with physiological measurements obtained from the electronic medical record (collected on the patient's arrival at the ED). Mortality data at 1, 2, 3, 7, 14, 30 and 90 days were also obtained from the electronic health records. The score values were subsequently used to analyze the predictive capacity of both scores for mortality at the different time points.

Primary data analysis
Categorical variables were represented by absolute values and percentages. Continuous variables were represented by median and interquartile range (IQR) since they did not follow a normal distribution. For the characterization of the total sample and to analyze the association between each independent variable and the primary outcome (90-day mortality), the Mann-Whitney U test or chi-squared test was performed as appropriate.
The discriminatory validity of the scores was assessed by the area under the receiver operating characteristic (ROC) curve (AUC), calculating in each case the p value of the hypothesis contrast. The graphs of the ROC curves show the confidence interval (95% CI) obtained by resampling (or bootstrapping) 2000 realizations. Finally, the specificity, sensitivity, positive predictive value, negative predictive value, positive likelihood ratio and negative likelihood ratio of the score obtained were calculated. With the objective of comparing ROCs, a Delong's test and a decision curve analysis were used. The effect of confounding factors (i.e. age, sex, and comorbidities, evaluated by the Charlson Age Comorbidity Index [CACI]) on the predictive value of both scores was assessed by a multivariate analysis. Data were analyzed using our own codes and base functions in R, version 4.0.3 (http://www.R-project.org; the R Foundation for Statistical Computing, Vienna, Austria).

Baseline characteristics
A total of 3183 patients met the inclusion criteria. After applying the exclusion criteria, the final sample for analysis consisted of 2961 patients with confirmed SARS-CoV-2 infection (see Supplementary Figure S1).

Scores' discrimination
The discrimination capacity of each score for 1-day and 90-day mortality was assessed analyzing the distribution of survivors and non-survivors and the predicted probability of mortality according to different values of the score (Figure 1). For NEWS, higher score values included a higher proportion of non-survivors and predicted a higher probability of mortality, both for 1-day (Figure 1(A)) and 90-day (Figure 1(B)) mortality. Moreover, lower score values predicted higher probability of death at 90 days than at 1 day. qCSI showed a similar distribution for 1-day ( Figure  1(C)) and 90-day (Figure 1(D)) mortality, although it provided lower probabilities of death than NEWS throughout the whole range of score values. Similar discrimination results at the other time points (2, 7, 14 and 30 days) can be found in the Supplementary Data S3.

Scores comparison
The predictive capacity of each score for mortality was assessed by ROC curves analysis and by decision curves. The comparison of results for both scores for mortality at 1 day and 90 days is shown in Figure 2. Similar comparisons were carried out for the intermediate time points (Supplementary Data S4). Although AUCs decreased with increasing times for both scores, AUC values were higher for the NEWS score than for qCSI throughout the whole range of time points (Table 2), indicating that NEWS has a better performance for mortality prediction. Finally, regarding statistical parameters related to AUCs, NEWS showed better sensitivity, positive predictive value, negative predictive value and positive likelihood ratio than qCSI, whereas its specificity and negative likelihood ratio were worse (Table 3). Finally, we carried out a multivariate analysis to assess the effect of putative confounding variables such as age, sex and comorbidities (CACI) on the predictive ability of both scores for mortality (Supplementary Data 5). For short mortality times (1 and 2 days) no effect was found. For 7-day mortality, being >89 years was a risk factor for both NEWS and qCSI and being male for NEWS. For longer times all confounding variables were found as risk factors for mortality.

Discussion
In the present study, the NEWS score showed a better ability to predict mortality in patients with COVID-19 than the qCSI score at all-time points analyzed during a 90-day follow-up period.
In an emergency situation, such as the current COVID-19 pandemic, it is critical to accurately predict the risk of clinical deterioration in order to balance provisions of life-saving care and to steward precious resources [14,21]. In this sense, early warning scores are reliable tools in such critical decisions [22]. For instance, NEWS is commonly used in several clinical situations [23][24][25][26] and has been validated for COVID-19 [7] . The qCSI, instead, was only internally validated [13], and there is limited experience in its implementation [27].
In the cohort of the present study, NEWS had an AUC of 0.825 for 1-day mortality, in line with recent studies [16,17,28,29], performing better than qCSI, which showed an AUC of 0.769. In the study by Haimovich et al. [13], where qCSI was established and validated, AUC of this score was 0.82, though the cohort used was smaller than ours. Importantly, in our study, the performance of both scores was not affected by age, sex or comorbidities up to 2 days, revealing their short-term potential. However, at longer mortality times, from 7 to 90 days, all the confounding factors were significantly related to mortality. This result was somehow expected, since at shorter times, initial deterioration of the patients may weigh more than intrinsic factors such as age sex and comorbidities.
Severe cases of SARS-CoV-2 are characterized by significant compromise of ventilatory function with breathlessness, desaturation and tachypnoea often requiring the administration of supplemental oxygen or invasive mechanical ventilation [30]. Both scores account (although with different weights) for ventilatory measurements, but NEWS additionally includes the cardio-vascular function (systolic blood pressure and heart rate), neurological function (Glasgow coma scale), and temperature. The addition of these parameters, a priori, makes it a more complete score than  qCSI. On the other hand, unlike NEWS, qCSI focuses only on the analysis of ventilatory function, which could be seen as an advantage, especially considering that it only requires three easy-to-obtain variables.
In critical circumstances with a volume of patients exceeding the operational capabilities of health systems, an appropriate selection of those cases with the highest risk of clinical deterioration is mandatory. In this sense, qCSI is a simple score, easy to apply, but with a worse performance than NEWS. However, NEWS has been implemented in multiple health systems, tested in a wide variety of clinical contexts, and has a high prognostic performance [31][32][33][34]. On the other hand, NEWS requires collection of more variables than the qCSI, an issue that should be considered a handicap during the pandemic. In short, both scores have strengths and weaknesses. However, based on more comprehensive scientific evidence, its implementation in multiple clinical environments, its robustness from a statistical point of view, and its better prognostic performance, NEWS could be considered a better suited score than qCSI for the EDs. This statement should be considered however with the warning message that this was a retrospective study; NEWS requires additional efforts [18,29,35,36], and it was not developed exclusively for COVID-19 patients.
During the first and second waves of the ongoing COVID-19 pandemic the initial patient/resource ratio was very high. This overload revealed the necessity of appropriate triage systems to apply the limited resources to critical patients. For instance, being classified as a high-risk patient (NEWS ! 7) is a strong predictor of early clinical impairment. Therefore, the use of scoring systems, such as NEWS, is critical to assist in initial triage, both at the scene and in the ED, helping to manage hospital and ICU admissions in a more efficient way, and to guide decisions on transfer to the ED. Nevertheless, its usefulness goes beyond this, because at extraordinary times like the present, it will allow us to know with certainty and in a simple way the short-term vital perspective of the patient. The availability of these scores will help the health system to better organize and manage limited resources, and also will allow us to inform the relatives of seriously ill patients of the possible outcome.
This study has several limitations. First, this is a retrospective study carried out during a pandemic. Due to the extreme burden of care at certain stages, the completion of clinical histories could have been compromised with an increase in the amount of missing data. To avoid bias, a multicenter study with a significant number of patients was planned. Second, due to the multiplicity of existing scores, a partial selection was made. Of all the early warning scores, NEWS is the one supported by the most scientific evidence, the highest degree of implementation, and by applications in diverse clinical contexts and pathologies including patients with COVID-19 [7,16,17]. Third, we avoided using scores that require analytical inputs (e.g. CURB-65) or parameters that are complex to quantify during the first care. Fourth, it should be highlighted that this study was developed by considering only those patients requiring EMS, which are arguably in a poorer clinical state as compared with those non-evacuated by ambulance. Finally, qCSI grounded its prognostic capacity on early clinical worsening during the first 24 h (necessity of high-flow oxygen, invasive or non-invasive mechanical ventilation, and death), whereas in the present study 90-day mortality was taken as the primary outcome variable. To solve this, we included 1-day mortality as an outcome, because qCSI was established and validated at this time point [13].
In summary, patients with SARS-CoV-2 infection can rapidly evolve to bilateral pneumonias with multiorgan disease; in these cases, an early identification of patients at high-risk of clinical deterioration should be prioritized. The NEWS score outperformed the qCSI score in predicting mortality at all studied time points, from 1 day to 90 days. The standardized use of early warning scores in patients evacuated by the EMS can aid in the complex decision-making process, assisting healthcare workers in the initial identification of the most severe patients and supporting the best allocation of resources.

Transparency declaration
The corresponding author on behalf of the other authors guarantee the accuracy, transparency and honesty of the data and information contained in the study, that no relevant information has been omitted and that all discrepancies between authors have been adequately resolved and described.
The following work has not been previously published and is not under consideration by any other scientific journal.

Author statement
Drs Mart ın-Rodr ıguez and L opez-Izquierdo had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: All authors. Acquisition, analysis, or interpretation of data: Delgado Benito, Mazas P erez-Oleaga and Garc ıa Villena. Drafting of the manuscript: Mart ın-Rodr ıguez and Sanz-Garc ıa. Final approval, critical revision of the manuscript for important intellectual content: All authors.